The following technique has been widely known. The technique, for example, searches a web page related to a keyword (a query) input by a user among web pages present on the Internet. Thus, the technique searches a content data item related to the query targeting a plurality of content data items including character information.
There is provided a search technique like this that scores and ranks content data items hit by search in accordance with a predetermined algorithm. In this case, a search result screen displays the content data items hit by the search in a display order in accordance with this ranking.
Here, a version upgrade to a new version is possibly performed on a search engine.
To upgrade the version of a search engine, instead of immediately replacing a search algorithm of a current version (hereinafter designated as a “current algorithm”) by a search algorithm of the new version (hereinafter designated as a “new algorithm”), it is preferable to replace the current algorithm with a new algorithm that has passed a preliminary test.
To make such preliminary test on the new algorithm a more practical test, the test can be conducted under an actual environment actually used by a general user. The test under the actual environment is, for example, conducted considering a use ratio of the new algorithm being substantially reduced, for example, the use ratio of the current algorithm to the new algorithm: 9:1. To facilitate confirmation of degree of improvement from the current algorithm, the test is conducted as a comparison test with the current algorithm.
The test under the actual environment uses an evaluation index related to a contribution to sales, for example, Click Through Rate (CTR) and ConVersion Rate (CVR), as a performance evaluation index for the search algorithm.